Thoughts about the JEV Model
The JEV has gone viral for a weird reason of cheap and fast, along with its “Reinforcement Learning for Calibration Decision” (RLCD). I want to comment some my questions and thoughts here.
Table of Contents
1. What Is JEV? What Does It Do?
JEV is a model introduced by TypeSafe AI. According to the official, it’s inspired by the two systems of thinking:
- system one. Fast, routine, unconcious decisions.
- system two. Slow, reasoning.
Recent LLMs specialize on system two tasks, i.e., coding agents, office agents. But for system one tasks, it’s more like classification rather than reasoning. So there seems to be an ability mismatch. In addition, the next-token-prediction nature of LLMs makes structured output more difficult.
So this is the gap that JEV tries to handle. As claimed by TypeSafe AI, it receives structured input, and produces structured output. The JEV model is trained with a so-called “Reinforcement Learning for Calibration Decisions” (RLCD).
2. Weird Hype
From my perspective, JEV seems to be just a hype. Hallucination is somehow intrinsic to the Transformers. As long as the backbone is a Transformer, the “no hallucination” claim is probably applicable to outputs.
In fact, we don’t even need a very complex archtecture to do this (I mean structured output). For example, we can simply utilize a LLM backbone but instead of autoregressive generation, we just need to train a classification or a regression head for “noul” tasks (yes-or-no questions); or just multiple vector similarity calculators for “choices” tasks with possibly softmax to ensure probability sum up to 1; … There’s already open-source competitor that can do the same thing1.
What I mean here is that, it’s very easy to simply output the score/confidence and let deterministic programs format scores into JSON. From the users’ point of view, Jev is equal to BERT + a JSON constructor. Then what’s the advantage of Jev?
3. RLCD
Let’s move on to RLCD. Obviously, it’s proposed to train their Jev model. But to be honest, the whole kind of task is so simple that I think any model can overfit on such tasks. Before RLCD is revealed, I would put a question mark here.
4. Conclusion
To conclude, I didn’t see any competitive edge over just adding a simple head and a deterministic constructor. It’s … weird to see so many people hyping this.